+++ /dev/null
-NOTE THIS STUFF IS OUT OF DATE! I'm working on merging some of these
-ideas into jhbuild for now.
-
-== The recipe set ==
-
-A recipe is similar to Bitbake's format, except just have metadata -
-we don't allow arbitrary Python scripts. Also, we assume
-autotools. Example:
-
-SUMMARY = "The basic file, shell and text manipulation utilities."
-HOMEPAGE = "http://www.gnu.org/software/coreutils/"
-BUGTRACKER = "http://debbugs.gnu.org/coreutils"
-LICENSE = "GPLv3+"
-LIC_FILES_CHKSUM = "file://COPYING;md5=d32239bcb673463ab874e80d47fae504\
- file://src/ls.c;startline=5;endline=16;md5=e1a509558876db58fb6667ba140137ad"
-SRC_URI = "${GNU_MIRROR}/coreutils/${BP}.tar.gz \
- file://remove-usr-local-lib-from-m4.patch \
- "
-DEPENDS = "gmp foo"
-
-Each recipe will output one or more artifacts.
-
-
-In GNOME, we will have a root per:
- - major version (3.0, 3.2)
- - "runtime", "sdk", and "devel"
- - Build type (opt, debug)
- - Architecture (ia32, x86_64)
-
-/gnome/root-3.2-runtime-opt-x86_64/{etc,bin,share,usr,lib}
-/gnome/root-3.2-devel-debug-x86_64/{etc,bin,share,usr,lib}
-/gnome/.real/root-3.2-runtime-opt-x86_64
-/gnome/.real/root-3.2-devel-debug-x86_64
-
-A "runtime" root is what's necessary to run applications. A SDK root
-is that, plus all the command line developer tools (gcc, gdb, make,
-strace). And finally the "devel" root has all the API-unstable
-headers not necessary for applications (NetworkManager.h etc.)
-
-Hmm, maybe we should punt developer tools into a Unix app framework.
-
-== Artifact ==
-
-An artifact is a binary result of compiling a recipe (there may be
-multiple). Think of an artifact as like a Linux distribution
-"package", except there are no runtime dependencies, conflicts, or
-pre/post scripts. It's basically just a gzipped tarball, and we
-encode metadata in the filename.
-
-Example:
-
-gdk-pixbuf-runtime,o=master,r=3.2-opt-x86_64,b=opt,v=2.24.0-10-g1d39095,.tar.gz
-
-This is an artifact from the gdk-pixbuf component. Here's a decoding of the key/value pairs:
-
-o: The origin of the build - there are just "master" and "local"
-r: The name of the root this artifact was compiled against
-b: The name of the build flavor (known values are "opt" and "debug")
-v: The output of "git describe".
-
-To build a root, we simply unpack the artifacts that compose it, and
-run "git commit".
-
-hacktree will default to splitting up shared libraries' unversioned .so
-link and header files into -devel, and the rest into -runtime.
-
-All binaries default to runtime.
-
-Local modifications ==
-
-A key point of this whole endeavour is that we want developers to be
-able to do local builds. This is surprisingly something not well
-supported by the Debian/Fedora's tools at least.
-
-=== Updating a root with a new local artifact ===
-
-Whenever you install a local artifact, if no "local" branch exists for
-that root, it's created.
-
-Let's say we're debugging gdk-pixbuf, tracking down a memory
-corruption bug. We've added a few printfs, and want to rerun things.
-GCC optimization is screwing us, so we build it in debug mode (-O0).
-The active root is root-3.2-opt.
-
-$ pwd
-~/src/gdk-pixbufroot
-$ echo $HACKTREE_ROOT
-/gnome/root-3.2-opt
-<hack hack hack>
-$ hacktree make debug
-<time passes, hopefully not too much>
-$ ls gdk-pixbuf*.tar.gz
-gdk-pixbuf-runtime,o=master,r=3.2-opt,b=debug,v=2.24.0-10-g1d39095,.tar.gz
-gdk-pixbuf-devel,o=master,r=3.2-opt,b=debug,v=2.24.0-10-g1d39095,.tar.gz
-gdk-pixbuf-manifests,o=master,r=3.2-opt,b=debug,v=2.24.0-10-g1d39095,.tar.gz
-$ hacktree install gdk-pixbuf*,o=master,r=3.2-opt,b=debug,v=2.24.0-10-g1d39095,.tar.gz
-<policykit auth dialog>
-
-Now here's where the cool stuff happens. hacktree takes
-/gnome/root-3.2-opt (the which is given in the r= above), and looks
-for the corresponding git branch (root-3.2-opt). Now hacktree notices
-there's no corresponding "local" branch, i.e. local-3.2-opt. One is
-created and checked out:
-
-# pwd
-/gnome/repo.git
-# git branch local-3.2-opt root-3.2-opt
-# git clone --branch local-3.2-opt /gnome/repo.git /gnome/.real-local-3.2-opt
-
-Now, the artifacts specified are overlaid:
-
-# cd /gnome/.real-local-3.2-opt
-# tar xvf
-
-Ok, now we need to remove old no longer shipped files from the root.
-Thus, we need a list of files corresponding to each original artifact,
-and to know which artifacts are in a root. Note above that one of the
-artifacts produced was "manifests". This contains files like:
-
-/meta/manifests/gdk-pixbuf-runtime.list
-/meta/manifests/gdk-pixbuf-devel.list
-
-Thus we diff the manifests, and clean up any leftover files.
-
-# git commit -a -m "Install artifact gdk-pixbuf-runtime,o=master,r=3.2-opt,b=debug,v=2.24.0-10-g1d39095,.tar.gz"
-# git checkout
-
-
OSTree
-========
+======
Problem statement
-----------------
stuck on whatever the distro provides.
Who is ostree for?
-------------------------------
+------------------
First - operating system developers and testers. I specifically keep
a few people in mind - Dan Williams and Eric Anholt, as well as myself
need to know how to interact with the installed distribution's grub.
This is an annoying but tractable problem.
-OSTree will allow efficiently parallel installing and downloading OS
-builds.
+First, we install a kernel+initramfs alongside the distribution's.
+Then, we have a "trampoline" ostree-init binary which is statically
+linked, and boot the kernel with init=/ostree/ostree-init. This then
+takes care of chrooting and running the init binary.
+
+An important note here is that we bind mount the real /home. This
+means you have your data. This also implies we share uid/gid, so
+/etc/passwd will have to be in sync. Probably what we'll do is have a
+script to pull the data from the "host" OS.
-An important note here is that we explicitly link /home in each root
-to the real /home. This means you have your data. This also implies
-we share uid/gid, so /etc/passwd will have to be in sync. Probably
-what we'll do is have a script to pull the data from the "host" OS.
+I've decided for now to move /var into /ostree to avoid sharing it
+with the "host" distribution, because in practice we're likely
+to hit incompatibilities.
-Other shared directories are /var and /root. Note that /etc is
-explicitly NOT shared!
+Do note however /etc lives *inside* the OSTree; it's presently
+versioned and readonly like everything else.
On a pure OSTree system, the filesystem layout will look like this:
|-- boot
|-- home
|-- ostree
+ | |-- var
| |-- current -> gnomeos-3.2-opt-7e9788a2
| |-- gnomeos-3.0-opt-393a4555
| | |-- etc
| |-- sys
| `-- usr
|-- root
- `-- var
Making this efficient
So we mentioned above there are:
- /gnomeos/root-3.0-opt
- /gnomeos/root-3.2-opt
+ /ostree/gnomeos-3.2-opt-7e9788a2
+ /ostree/gnomeos-3.2-opt-393a4555
There is also a "repository" that looks like this:
- .ht/objects/17/a95e8ca0ba655b09cb68d7288342588e867ee0.file
- .ht/objects/17/68625e7ff5a8db77904c77489dc6f07d4afdba.meta
- .ht/objects/17/cc01589dd8540d85c0f93f52b708500dbaa5a9.file
- .ht/objects/30
- .ht/objects/30/6359b3ca7684358a3988afd005013f13c0c533.meta
- .ht/objects/30/8f3c03010cedd930b1db756ce659c064f0cd7f.meta
- .ht/objects/30/8cf0fd8e63dfff6a5f00ba5a48f3b92fb52de7.file
- .ht/objects/30/6cad7f027d69a46bb376044434bbf28d63e88d.file
+ /ostree/repo/objects/17/a95e8ca0ba655b09cb68d7288342588e867ee0.file
+ /ostree/repo/objects/17/68625e7ff5a8db77904c77489dc6f07d4afdba.meta
+ /ostree/repo/objects/17/cc01589dd8540d85c0f93f52b708500dbaa5a9.file
+ /ostree/repo/objects/30
+ /ostree/repo/objects/30/6359b3ca7684358a3988afd005013f13c0c533.meta
+ /ostree/repo/objects/30/8f3c03010cedd930b1db756ce659c064f0cd7f.meta
+ /ostree/repo/objects/30/8cf0fd8e63dfff6a5f00ba5a48f3b92fb52de7.file
+ /ostree/repo/objects/30/6cad7f027d69a46bb376044434bbf28d63e88d.file
Each object is either metadata (like a commit or tree), or a hard link
to a regular file.
system, or the old one. There is no "Please don't turn off your
computer". We do this by simply using a symbolic link like:
-/gnomeos -> /gnomeos-e3b0c4429
+/ostree/current -> /ostree/gnomeos-3.4-opt-e3b0c4429
-Where /gnomeos-e3b0c4429/ has the full regular filesystem tree with
-usr/ etc/ directories as above. To upgrade or rollback (there is no
+Where gnomeos-e3b0c4429 has the full regular filesystem tree with usr/
+etc/ directories as above. To upgrade or rollback (there is no
difference internally), we simply check out a new tree into
-/gnomeos-b90ae4763 for example, then swap the symbolic link, then
-remove the old tree.
+gnomeos-b90ae4763 for example, then swap the "current" symbolic link,
+then remove the old tree.
-But does this mean you have to "reboot" for OS upgrades? Very likely,
+But does this mean you have to reboot for OS upgrades? Very likely,
yes - and this is no different from RPM/deb or whatever. They just
-typically lie to you about it =) But read on.
-
-Let's consider a security update to a shared library. We can download
-the update to the repository, build a new tree and atomically swap it
-as above, but what if a process has the old shared library in use?
-
-Here's where we will probably need to inspect which processes are
-using the library - if any are, then we need to trigger either a
-logout/login if it's just the desktop shell and/or apps, or a
-"fastboot" if not. A fastboot is dropping to "init 1" effectively,
-then going back to "init 5".
+typically lie to you about it =)
+
+A typical model with RPM/deb is to unpack the new files, then use some
+IPC mechanism (SIGHUP, a control binary like /usr/sbin/apachectl) to
+signal the running process to reload. There are multiple problems
+with this - one is that in the new state, daemon A may depend on the
+updated configuration in daemon B. This may not be particularly
+common in default configurations, but it's highly likely that that
+some deployments will have e.g. apache talking to a local MySQL
+instance. So you really want to do is only apply the updated
+configuration when all the files are in place; not after each RPM or
+.deb is installed.
+
+What's even harder is the massive set of race conditions that are
+possible while RPM/deb are in the process of upgrading. Cron jobs are
+very likely to hit this. If we want the ability to apply updates to a
+live system, we could first pause execution of non-upgrade userspace
+tasks. This could be done via SIGSTOP for example. Then, we can swap
+around the filesystem tree, and then finally attempt to apply updates
+via SIGHUP, and if possible, restart processes.
Configuration Management
------------------------
gcc. We then import that into an OSTree branch
e.g. "bases/gnomeos-3.4-yocto-i686-devel".
-Now we also assume that you have ostree installed on the host build
-system via e.g. jhbuild or RPM if doing a cross build. The core
-ostbuild tool can then chroot into a checkout of the Yocto base, and
-start generating artifacts.
+We also have a Yocto recipe "ostree-native" which generates (as you
+might guess) a native binary of ostree. That binary is used to import
+into an "archive mode" OSTree repository. You can see it in
+$builddir/tmp/deploy/images/repo.
-Each generated artifact is committed to an OSTree branch like
-"artifacts/gnomeos-3.4-i686-devel/libxslt/master/runtime". Then, a
-"compose" process merges together the individual filesystem trees into
-the final branches (e.g. gnomeos-3.4-i686-devel), and the process
-repeats.
+Now that we have an OSTree repository storing a base filesystem, we
+can use "ostbuild" which uses "linux-user-chroot" to chroot inside,
+run a build on a source tree, and outputs binaries, which we then add
+to the build tree for the next module, and so on.
ostbuild details
--------------------
+----------------
The simple goal of ostbuild is that it only takes as input a
"manifest" which is basically just a list of components to build. A
to review all of the code that goes in, and to efficiently store
source code updates.
-For GNOME, tarballs are mostly pointless - it's easy enough to just
-run autogen.sh. However there are two challenges:
+The result of a build of a component is an OSTree branch like
+"artifacts/gnomeos-3.4-i686-devel/libxslt/master". Then, a "compose"
+process merges together the individual filesystem trees into the final
+branches (e.g. gnomeos-3.4-i686-devel).
-1) Tarballs for modules which self-build-depend may include
- pre-generated files. For example - flex's tarball includes a
- generated .c file for the parser. For these, we can either move
- the module build to the Yocto level (thus giving a convenient way
- to pull in host files), or possibly add the ability to
- hardlink/copy in host binaries to ostbuild.
-
-2) Tarballs which include translations pulled from a different
- location. For example - bison. For these, we basically have to
- maintain our own git repositories.
-
-
-
-building
---------
+Doing a full build on your system
+---------------------------------
srcdir=/src
builddir=/src/build
projects / ostree.git / commitdiff